Identifying Sources of Disagreement: Generalizability Theory in Manual Annotation Studies

Identifying Sources of Disagreement: Generalizability Theory in Manual Annotation Studies

Many annotation projects have shown that the quality of manual annotations often is not as good as would be desirable for reliable data analysis. Identifying the main sources responsible for poor annotation quality must thus be a major concern. Generalizability theory is a valuable tool for this purpose, because it allows for the differentiation and detailed analysis of factors that influence a...

Generalizability of Adoption Studies

How the probands are recruited also can affect the quality of a study’s conclusions. One potential source of bias is the influence of environmental factors on the selection of proband adoptees in the adoptees’ family method. For example, psychological or social problems in an adoptive family may contribute to the adoptee’s psychopathology. Simultaneously, these problems may prompt the family an...

Generalizability Theory: Overview

36 Generalizability Theory

Generalizability (G) theory is a statistical theory for evaluating the dependability (or reliability) of behavioral measurements (Cronbach, Gleser, Nanda, & Rajaratnam, 1972; see also Brennan, 2001; Shavelson & Webb, 1991). G theory permits the researcher to address such questions as: Is the sampling of tasks or judges the major source of measurement error? Can I improve the reliability of the ...

Crowdsourcing Disagreement for Collecting Semantic Annotation

This paper proposes an approach to gathering semantic annotation, which rejects the notion that human interpretation can have a single ground truth, and is instead based on the observation that disagreement between annotators can signal ambiguity in the input text, as well as how the annotation task has been designed. The purpose of this research is to investigate whether disagreement-aware cro...